1 Interpretations of Probability

What does "probability" mean in the real world?

• Long-run frequency over repeated trials.
  Ex, repeatedly flipping a coin, shooting electrons at a double slit.
• systematic random sampling from a population.
  Ex, survey of ${\displaystyle 500}$ random voters, random assignment to treatment/control group. Randomness comes from experimenters' actions.
• Subjective uncertainty about an outcome.
  Ex, chance that Trump is elected, ${\displaystyle 100}$ th digit of ${\displaystyle \pi }$ is ${\displaystyle 5}$. Could be broad intersubjective agreement.

2 Where does Prior ${\displaystyle {\displaystyle \mathrm{\Lambda }}}$ Come From?

2.1 Subjective Beliefs

Subjective beliefs bring all relevant information to bear straightforward interpretation of posterior. But posterior is therefore subjective.

2.2 "Objective" or "Vague" Prior

Using default prior removes subjectivity. Like flat prior ${\displaystyle \lambda (\theta ){\propto }_{\theta }1}$ on ${\displaystyle \mathrm{\Theta }}$.

We also have Jeffrey's prior ${\displaystyle \lambda (\theta ){\propto }_{\theta }|J(\theta ){|}^{\frac{1}{2}}}$ (recall Fisher information). In this case, recall this formula, we have $$\frac{\sqrt{D_{\mathrm{KL}}(p_{\theta }||p_{{\theta }^{\ast }})}}{|{\theta }^{\ast }-\theta |}\approx \frac{\sqrt{J(\theta )}}{\sqrt{2}}.$$ Then when ${\displaystyle \epsilon }$ is small, $$\mathrm{\Lambda }([\theta ,\theta +\epsilon ])\approx \epsilon \lambda (\theta ){\propto }_{\theta }\sqrt{2D_{\mathrm{KL}}(p_{\theta }||p_{\theta +\epsilon })}.$$
${\displaystyle \mathrm{\Lambda }(\theta )}$ has higher density when ${\displaystyle p_{\theta }}$ is "changing faster".

Take coin tossing as an example:

2.2.1 Intersubjective Agreement

Data may effectively rule out most ${\displaystyle \theta }$ values. This will make posterior uncontroversial.

2.2.2 Gaussian Sequence Model

Let ${\displaystyle X|\theta \sim {\mathcal{N}}_d(\mu ,I_d)}$, ${\displaystyle \mu \in {\mathbb{R}}^d}$. Jefferey's prior is flat: ${\displaystyle \lambda (\mu ){\propto }_{\mu }I}$.^[1]

So ${\displaystyle \lambda (\mu |X)={\mathcal{N}}_d(X,I_d)\Rightarrow \mathbb{E}[\mu |X]=X}$. This is the same as UMVUE.

For ${\displaystyle p^2=||\mu |{|}^2}$, recall ${\displaystyle \mu \sim {\mathcal{N}}_d(X,I_d)\Rightarrow \mathbb{E}[||\mu |{|}^2|X]=||X|{|}^2+d}$, and note that ${\displaystyle {\delta }_{\mathrm{UMVU}}(X)=||X|{|}^2-d}$, so ${\displaystyle {\delta }_{\lambda }(x)={\delta }_{\mathrm{UMVU}}(x)+2d}$.
Now apply bias-variance tradeoff: $$\mathrm{MSE}(\theta ;{\delta }_{\lambda })={\mathrm{Var}}_{\theta }({\delta }_{\lambda })+{\mathrm{Bias}}_{\theta }({\delta }_{\lambda }{)}^2={\mathrm{Var}}_{\theta }({\delta }_{\mathrm{UMVU}})+4d^2.$$
Examine Jeffrey's prior again: $$\begin{array}{rl}& \mathbb{P}(p^2\le t)=\mathrm{Vol}(\text{Ball of radius }\sqrt{t})=\mathrm{const}(d)\cdot t^{\frac{d}{2}}\\ \Rightarrow & \lambda (p^2){\propto }_{p^2}(p^2{)}^{\frac{d}{2}-1}=p^{d-2},\end{array}$$ which grows rapidly. This shows prior "expects" ${\displaystyle p^2}$ to be large.

2.3 Prior or Concurrent Experience

2.3.1 Flexibility of Bayes

Given any ${\displaystyle \mathrm{\Lambda },\mathcal{P},\mathcal{L},g(\theta )}$, ${\displaystyle {\delta }_{\mathrm{\Lambda }}}$ is defined straightforwardly by $${\delta }_{\mathrm{\Lambda }}(x)=\arg \underset{d}{min}\int \mathcal{L}(\theta ,d)\lambda (\theta |x)\mathrm{d}\theta .$$ (Recall here)So problem is reduced to (possibly hard) computation. So posterior is "one stop shop" for all answers. There is no need for

• Special family structure (exponential family, complete sufficient statistic, etc.)
• special estimator (like U-estimable)
• convex or nice ${\displaystyle \mathcal{L}}$

So it is highly expressive modeling and estimation. (The caveat is the limitation by ability to do computations)

2.4 Convenience Priors

We can choose conjugate or other "nice" priors, so computations will be much faster, especially in high dimensions.

3 Hierarchical Bayes

Full power of Bayes is realized in large, complex problems with repeat structure, allowing us to pool information across many observations.

3.1 Gaussian Hierarchical Model

Now suppose $$\begin{array}{rl}{\tau }^2& \sim {\lambda }_0,\\ {\theta }_i|{\tau }^2& \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\tau }^2),i\le d,\\ X_i|{\tau }^2,\theta & \stackrel{\mathrm{i}.\mathrm{n}.\mathrm{d}}{\sim }\mathcal{N}({\theta }_i,1).\end{array}$$
Calculate the posterior mean: $$\begin{array}{rl}\delta (x_i)& =\mathbb{E}[{\theta }_i|x]\\ & =\mathbb{E}[\mathbb{E}[{\theta }_i|x,{\tau }^2]|X]\\ & =\mathbb{E}\left[\frac{{\tau }^2}{1+{\tau }^2}{X}_i|X\right]\cdot X_i.\end{array}$$

Linear shrinkage estimator, Bayes-optimal shrinkage estimated from data.

Likelihood for ${\displaystyle {\tau }^2}$: marginalized over ${\displaystyle {\theta }_i}$: $$\begin{array}{rl}& X_i|{\tau }^2\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,1+{\tau }^2)\\ \Rightarrow & \frac{1}{d}||X|{|}^2\sim \frac{1+{\tau }^2}{d}{\chi }_d^2\sim (1+{\tau }^2,\frac{2(1+{\tau }^2)}{d}).\end{array}$$
Define ${\displaystyle \zeta ({\tau }^2)=\frac{1}{1+{\tau }^2}}$, (amount of shrinkage) so ${\displaystyle \delta (X)=(1-\mathbb{E}[\zeta |X])X_i}$.

Conjugate prior: $$\begin{array}{rl}\zeta & \sim \frac{1}{s^2}{\chi }_k^2=\mathrm{\Gamma }(\frac{k}{2},\frac{2}{s^2})=\frac{(s^2{)}^{\frac{k}{2}}}{\mathrm{\Gamma }\left(\frac{k}{2}\right)}{\zeta }^{\frac{k}{2}-1}{\mathrm{e}}^{-\frac{s^2\zeta }{2}}\\ \Rightarrow \zeta |||X|{|}^2& {\propto }_{\zeta }{\zeta }^{\frac{k+d}{2}-1}{\mathrm{e}}^{-(s+||X|{|}^2)\zeta }\sim \frac{{\chi }_{k+d}^2}{s+||X|{|}^2}\\ \Rightarrow \mathbb{E}[\zeta |||X|{|}^2]& =\frac{k+d}{s+||X|{|}^2}.\end{array}$$
"pseudo-data"${\displaystyle =Y_1,\cdots ,Y_k}$ with ${\displaystyle ||Y|{|}^2=s^2}$. We might want to truncate prior to ${\displaystyle [0,1]}$ if ${\displaystyle d}$ is small.
#？

3.2 Graphical Form

These are directed graphical models. Implies the distribution may be factorized with one factor for each vertex in ${\displaystyle (V,E)}$: $$p(z_1,\cdots ,z_{|V|})=\prod _{i=1}^{|V|}{p}_i(z_i|z_{p_a(i)}),$$ ${\displaystyle p_a(i)=\{j:j\to i\}}$. For this model, $$p({\tau }^2,{\theta }_1,\cdots ,{\theta }_m;x_1,\cdots ,x_m)=p({\tau }^2)\prod _{i=1}^{m}p({\theta }_i|{\tau }^2)\prod _{i=1}^{n}p(x_i|{\theta }_i).$$